Screen sharing fails sometimes, freezing the iMac I'm sharing. what am I doing wrong? only fix is to restart that Mac?

Most of the time it works fine. This is one scenario when it fails:

iMac #1 is being used by someone else. I start screen sharing from my older iMac(#2) to my account on #1, not disturbing the other user in other room. It works funny when I do this, but it does work o-k, in a pinch. other user at some point logs off or leaves, and I try to re-login or do my regular screen sharing routine(Turn on #1, put it on login/users screen, activate screen sharing from #2, log into my account...)to get better screen sharing functionality. but the screen is dark & blank, everything has disappeared except for the cursor arrow. can't get a login window or users list. Only thing we can do at this point is to shut down the Mac and restart, losing some of our work.

What am I doing wrong? How can I avoid having to restart the Mac?